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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the 
fee set forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since 
this application is eligible for continued examination under 37 CFR 1.114, and the fee 
set forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office 
action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
December 2 nd , 2005 has been entered. 

Response to Arguments 

2. Applicants amendments filed December 2 nd , 2005 have been entered and 
made of record. 

3. Claims 1 and 3 have been amended. No claims have been canceled and 
Claims 1-1 1 are now pending. 

4. Applicants amendments and arguments have been fully considered but 
are not found fully persuasive for at least the following reasons: 

An after-final request for reconsideration was filed on August 1 1 th 2005 and an 
advisory office action was filed on August 23 rd 2005 in response to the after-final 
request addressing the arguments. It appears that the same basic arguments in scope 
are presented again by Applicant in response to the final office action with no 
consideration of the explanation presented by the Office in the advisory office action 
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filed August 23 rd 2005. The test of the advisory action is presented below for the 
Applicants convenience: 

Applicant argues that the reference of Davison does not disclose all of the 
limitations of the independent claims. Applicant focuses on the feature of the resolution 
map and selecting of a pixel depending on its resolution and on that of the pixels of 
other of the sequence matched with this pixel. Examiner submits that the cited passage 
of Davison referring to the shift or distance between pixels in steps S558-S560 
interpreted broadly still reads o the applicants definition and disclosure of 3D resolution 
in the present applicaton. Davison discloses determining shift which is effectively 
seperation distance of points in 3D space which is effectively the resolution of the 3D 
image coordinates. 

Further applicant is directed to a similar disclosure in Davison in column 21, 
lines 14-26. Davison discloses using grid squares of different sizes to determine 
density of points in the respective images to be matched. This sounds like exactly the 
Applicants definition of resolution on page 5, lines 14-21 of the specification. Applicant 
further argues that "Davison et al. neither disclose nor suggest matching of a pixel of a 
current image with a pixel of another image of the sequence, pixels relating to one and 
the same point of the 3D scene, by projecting the pixel of the current image onto the 
other image." Examiner points to column 23, lines 1-18 where Davison discloses the 
matching of points in the two images and forming a measurement matrix M to be used 
in forming the 3D model. Applicant further alleges that "Davison et al. are not 
concerned with the use of the 3D model and mathcing images to identify redundancy as 
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in the present claimed invention. " Examiner submits that this is exactly Davison's goal 
(column 5, lines 47-51). Applicant's remarks with regard to the 103 rejections appear to 
further argue that Davison does not disclose the invention as claimed in the 
independent claims. Examiner submits that the reference of Davison applies and that 
the 103 rejections still apply. 

The arguments now presented are addressed below. 

In response to the Applicants remarks with regard to the elements being 
"arranged as required by the claim", Examiner would first like to point out that what is 
claimed in claim 1 is a method comprising a certain number of steps. Therefore as 
broadly and reasonably interpreted the cited method of Davison need only read on the 
steps of the method as claimed. Claim 1 has been amended and now changes the 
scope of the invention. Below is an explanation of why the claims still read on the 
Reference of Davison as reasonably broadly interpreted. 

With regard to claim 1, Davison discloses a method for constructing a 3D scene 
model by analyzing image sequences, each image corresponding to a viewpoint defined 
by its position and its orientation (Fig. 29A and Fig.3 steps S10-S14), comprising the 
following steps: 

Calculating, for each image, a depth map corresponding to the depth, in 3D 
space, of pixels of the image (column 7, lines 34-45). Here Davison disclose that after 
an initial matching step, the camera orientation and the positions are determined with 
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regard to the matched points in relative form. This information is interpreted as depth 
maps because the points in the image are determined relative to their capture 
viewpoints. Furthermore it is a common practice that depth information be found for 
points or pixels through the process of stereo image correspondence using two or more 
images to determine depth. The way in which the depth maps for each image are 
determined are not disclose in the present specification and is therefore not enabled. 
On page 4, line 20 of the specification it is stated "An ad hoc processing provides for 
each image, a depth map as well as the position and orientation of the corresponding 
viewpoint." On page 5, lines 3-6 it is stated that "Available at system input, for ach 
image sequence, is a depth map as well as the position and the orientation of the 
corresponding viewpoint." There is nothing in the disclosure enabling the claimed 
feature of "calculating for each image, a depth map corresponding to the depth, in 3D 
space of pixels of the image." It should have been explained how this is done for each 
image if this is an important part of the operation of the invention. A 1 12 first paragraph 
rejection is accordingly applied below. 

Davison further discloses calculating, for each image, a resolution map 
corresponding to the 3D resolution of said pixels of the image from the depth map 
(column 7, lines 40-50). Davison discloses that further features are matched once the 
depth information is found and using the pre-matched points. A resolution map 
interpreted reasonably broadly is simply an image having a resolution. The resolution of 
the points is interpreted as being necessary to matching points or features. There must 
be a certain resolution in working with the images in terms of separation and how many 
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pixels or data points are needed per area/volume in order to perform matching. That is 
the point shown in Davison at figures 45A-47. There must be a resolution determination 
in order to match the points and represent points in 3D space and the figures illustrate 
that. 

Davison further discloses matching a pixel of a current image with a pixel of 
another image of the sequence, said pixels relating to one and the same point of the 3D 
scene, by projecting the pixel of the current image onto the other image (Fig. 3, step S8, 
column 7, lines 39-45 and column 10, lines 53-61, column 18, lines 34-59 and/or 
column 33, lines 38-53 and/or column 41, lines 40-55 and column 42, lines 31-54). 

Davison further discloses selecting a pixel of the current image depending on its 
resolution and on that of the pixels of other images of the sequence matched with said 
pixel (Notice steps 558-560 of Davidson et al. Fig. 44a. Here, discarding points of 
excessive inter-point distance (i.e. low 3D resolution), results in the selection of a set of 
points, and correspondingly a set of pixels, used in the subsequent construction). 

Davison further discloses constructing a 3D model from the selected pixels (Fig. 
3, steps S12-S 14). 

Now it should be clear from the explanation of Davison that the substance of 
claim 1 is disclosed. The response of the 103 rejections relying on Davison as the 
primary reference should be made clear in view of the above discussion. Therefore the 
rejections of claims 1, 2, 4, 5 and 9-11 are maintained. 



Application/Control Number: 09/831 ,992 Page 7 

Art Unit: 2623 

Claim Rejections - 35 USC §112 

5. Claim 1 is rejected under 35 U.S.C. 112, first paragraph, as failing to 
comply with the enablement requirement. The claim(s) contains subject matter which 
was not described in the specification in such a way as to enable one skilled in the art to 
which it pertains, or with which it is most nearly connected, to make and/or use the 
invention. 

Claim 1 claims the step of: 

Calculating for each image a depth map corresponding to the depth, in 3D space, 
of pixels of the image. 

However it is not explained as to how those depth maps are calculated in the 
specification or that they are calculated at all. 

On page 4, line 20 of the specification in the section entitled "Brief Description 
About the Drawings", it is stated "An ad hoc processing provides for each image, a 
depth map as well as the position and orientation of the corresponding viewpoint." On 
page 5, lines 3-6 it is stated that "Available at system input, for each image sequence, is 
a depth map as well as the position and the orientation of the corresponding viewpoint." 
It is unclear as to whether the calculating depth maps for each image is within the scope 
of the invention. There is nothing in the disclosure enabling the claimed feature of 
"calculating for each image, a depth map corresponding to the depth, in 3D space of 
pixels of the image." It should be explained how this is done for each image if this is 
part of the operation of the invention. 
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It is a common practice that depth information be found for points or pixels 
through the process of stereo image correspondence using two or more images to 
determine depth. The way in which the depth maps for each image are calculated is not 
disclosed in the present specification and is therefore not enabled. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

6. Claims 1, 9 and 11 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Davidson et al. 

With regard to claim 1 , Davison discloses a method for constructing a 3D scene 
model by analyzing image sequences, each image corresponding to a viewpoint defined 
by its position and its orientation (Fig. 29A and Fig. 3 steps S10-S14), comprising the 
following steps: 
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Calculating, for each image, a depth map corresponding to the depth, in 3D 
space, of pixels of the image (column 7, lines 34-45). Here Davison disclose that after 
an initial matching step, the camera orientation and the positions are determined with 
regard to the matched points in relative form. This information is interpreted as depth 
maps because the points in the image are determined relative to their capture 
viewpoints. 

Davison further discloses calculating, for each image, a resolution map 
corresponding to the 3D resolution of said pixels of the image from the depth map 
(column 7, lines 40-50). Davison discloses that further features are matched once the 
depth information is found and using the pre-matched points. The resolution of the 
points is interpreted as being necessary to matching points or features. There must be 
a certain resolution in working with the images in terms of separation and how many 
pixels or data points are needed per area/volume in order to perform matching. That is 
the point shown in Davison at figures 45A-47. There must be a resolution determination 
in order to match the points and represent points in 3D space and the figures illustrate 
that. 

Davison further discloses matching a pixel of a current image with a pixel of 
another image of the sequence, said pixels relating to one and the same point of the 3D 
scene, by projecting the pixel of the current image onto the other image (Fig. 3, step S8, 
column 7, lines 39-45 and column 10, lines 53-61, column 18, lines 34-59 and/or 
column 33, lines 38-53 and/or column 41, lines 40-55 and column 42, lines 31-54). 
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Davison further discloses selecting a pixel of the current image depending on its 
resolution and on that of the pixels of other images of the sequence matched with said 
pixel (Notice steps 558-560 of Davidson et al. Fig. 44a. Here, discarding points of 
excessive inter-point distance (i.e. low 3D resolution), results in the selection of a set of 
points, and correspondingly a set of pixels, used in the subsequent construction). 

Davison further discloses constructing a 3D model from the selected pixels (Fig. 
3, stepsS12-S14). 

Now it should be clear from the explanation of Davison that the substance of 
claim 1 is disclosed. The response of the 103 rejections relying on Davison as the 
primary reference should be made clear in view of the above discussion. Therefore the 
rejections of claims 1 , 2, 4, 5 and 9-1 1 are maintained. 

The following is in regard to Claim 9. As shown above, Davidson et al. disclose a 
3D model construction method that sufficiently conforms to the method of claim 1 . In the 
method of Davidson et al., the pixel of the other image is the pixel closest to the 
projection point on this other image. See, for example, Davidson et al. Fig. 35. The 3D 
model construction method of Davidson et al. thus sufficiently conforms to the method 
proposed by the Applicant in claim 9. Therefore, the teachings of Davidson et al. 
anticipate the method of claim 9. 

The following is in regard to Claim 11. As shown above, Davidson et al. disclose 
a 3D model construction method that sufficiently conforms to the method of claim 1 . 
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Davidson et al. suggest the usage of the derived model in virtual reality applications 
(e.g. ones that employ VRML - Davidson et at. column 6, lines 3-5). The following 
aspects of claim 1 1 are inherent to virtual reality applications: 

(1 1 .ad) Creating images as a function of the movement of the viewpoint. 

(1 1 .bD) These images are of different viewpoints of a 3D model. 
Therefore, a virtual-reality application utilizing a model(s) obtained according to the 
method of Davidson et al., would constitute a method of navigation in a 3D scene 
consisting of: 

(1 1 .a.) Creating images as a function of the movement of the viewpoint. 
(1 1 .6) These images are of different viewpoints of the 3D model(s) 
derived according to Claim 1 . 
This is clearly in accordance with the method proposed in claim 1 1 . 



Rejections Under 35 U.S.C. § 103(a) 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed 
or described as set forth in section 1 02 of this title, if the differences between the 
subject matter sought to be patented and the prior art are such that the subject 
matter as a whole would have been obvious at the time the invention was made 
to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was 
made. 
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10. Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Davidson et al., in view of Azarbayejani et al. (U.S. Patent 5,511,153). 

The following is in regard to Claim 2. As shown above, Davidson et al. disclose a 
3D model construction method that sufficiently conforms to the method of claim 1. 
Clearly, the points selected according to the method of Davidson et al. constitute one or 
more regions. Furthermore, as stated above, the method of Davidson et al. selects 
pixels depending on the resolution (see the discussion relating to step (1.d) above). 
However, Davidson et al. do not expressly show or suggest: 

(2. a.) Calculation and allocation of weights to the pixels of the image 
depending on: 

1 . whether or not they belong to the regions 

2. on the geometrical characteristics of the regions to which 
they belong in the image. 

(2.b.) A selection of the pixels performed depending on the weight values 

assigned to the pixels. 
Azarbayejani et al. 1 , on the other hand, disclose a method for constructing a 3D 
scene model by analyzing image sequences, wherein: 

(2. an.) Weights (e.g. i/(i+z c /?) in Azarbayejani et al. equation (2) or □ in 
Azarbayejani et al. equation (3)) are calculated and allocated to the 



1 Note that the various equations of Azarbayejani et al. will be referred to as (1), (2), (3) and so 
on, beginning with the equation listed in Azarbayejani et al. column 2, line 40. 
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pixels of the image. See Azarbayejani et al. equations (2) and (3), 
for example. The weights D g represent the geometric 
characteristics or structure of an image feature (i.e. region) to 
which the pixels belong (Azarbayejani et al. column 6, lines 5-13) 
and are indicative of whether or not they belong to the feature 
(Azarbayejani et al. column 5, lines 54-60). 
(2.bCL) A selection of the pixels performed depending on the weight values 
assigned to the pixels (Azarbayejani et al. column 10, lines 52-61). 
The teachings of Davidson et al. and Azarbayejani et al. are combinable because 
they are analogous art. Specifically, the teachings of both are directed to 3D model 
generation from image sequences. Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to perform 
feature tracking (or matching, using Davidson et al/s nomenclature) in the method of 
Davidson et al. according to steps (2.aD.)-(2.bD.) of Azarbayejani et al. One would have 
been motivated to do so because of the accuracy of Azarbayejani et al/s method. 
Combining the teachings of Davidson et al. and Azarbayejani et al. yields a method, in 
accordance with claim 1 , such that: 

(2.a.) Calculation and allocation of weights to the pixels of the image 
depending on: 

1 . whether or not they belong to the regions 

2. on the geometrical characteristics of the regions to which 
they belong in the image. 
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(2.b.) A selection of the pixels performed depending on the resolution 
and weight values assigned to the pixels. 
Therefore, such a method would conform to the method proposed in claim 2. 

11. Claims 4-5 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Davidson et al., in view of McAllister et al. ("Real-Time Rendering Techniques of 
Real World Environments"). 

The following is in regard to Claim 4. As shown above, Davidson et al. disclose a 
3D model construction method that sufficiently conforms to the method of claim 1 . 
Davidson et al., however, do not expressly show or suggest a partitioning of the images 
of the sequence characterized by: 

(4. a.) Identifying, for a current image, the images whose corresponding 
viewpoints have an observation field possessing an intersection 
with the observation field relating to the current image. 
(4.b.) Forming a list of images associated therewith. 
(4.c.) The list contains images of the sequence that will be used in the 

matching of the pixels of the current image. 
McAllister et al. disclose an image-based modeling method that generates a 3D 
model of an object from a multitude of 2D images (McAllister et al. Abstract). The 
method entails a partitioning of the images (i.e. into tiles) of the sequence (McAllister et 
al. Section 4.5, paragraph 1) that includes the following: 
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(4,b.) Forming a list of images associated therewith (McAllister et al. 

Section 4.6, paragraph 2). 
(4.c.) The list contains images of the sequence that will be used in the 

matching of the pixels of the current image (McAllister et al. 

Section 4.6, paragraph 2, in particular, lines 4-7). 

Note that: 

(4.a.) Identifying, for a current image, the images whose corresponding 
viewpoints have an observation field (i.e. view frustum - McAllister 
et al. Section 4.5 paragraph 1, line 4) possessing an intersection 
with the observation field relating to the current image, 
is inherent to the view-frustum culling of McAllister's et al.'s method. Those view- 
frustums that are culled correspond to images that sample surfaces which are disjoint 
from the surfaces imaged by the retained view-frustums. One can conclude from this 
that the retained and culled view-frustums are divergent. Thus, the view-frustum culling 
represents an implicit identification of intersecting view-frustums (i.e. retained view- 
frustums). That is, the retained view-frustums are those that intersect. 

The teachings of McAllister et al. and Davidson et al. are combinable because 
they are analogous art. Specifically, the teachings of both are directed toward the 
construction of 3D models of scenes from a collection of 2D images. Therefore, it would 
have been obvious to one of ordinary skill in the art, at the time of the applicant's 
claimed invention, to integrate the partitioning scheme of McAllister et al. into the 
method of Davidson et al. The motivation to do so would have been to and eliminate 
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redundant views and imagery from the data set (McAllister et al. Section 4.5, paragraph 
1 , sentence 3), thereby, improving rendering efficiency (McAllister et al. Section 4.6, last 
sentence) and storage overhead. Combining the teachings of McAllister et al. and 
Davidson et al., in this manner, yields a 3D modeling method that conforms 
substantially to that of claim 4. 

The following is in regard to Claim 5. As just shown, the teachings of McAllister et 
al. and Davidson et al. can be combined to yield a method that adequately satisfies the 
limitations of claim 4. Furthermore, the partitioning of the images of the sequence, 
according to the teachings of McAllister, is performed by removing, from the list 
associated with an image, the images which possess too few pixels corresponding to 
those of the current image (McAllister et al. Section 4.6, paragraph 2). Therefore, 
combining the teachings of McAllister et al. and Davidson et al., in the manner 
discussed above, yields a 3D modeling method that conforms substantially to that of 
claim 5. 

12. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Davidson et al., in view of La Roux et al. ("An Overview of Moving Object Segmentation 
in Video Images", IEEE, 1991). 

The following is in regard to Claim 10. As shown above, Davidson et al. disclose 
a 3D model construction method that sufficiently conforms to the method of claim 1 . As 
mentioned above, Davidson et al. attempt to find pixels in the sequence of images that 
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correspond to the same (or approximately the same) 3D point of the imaged scene or 
object. The following problems associated with deriving stereo correspondences 
between images of moving objects are well known. Besides potentially inducing 
undesirable motion blur in the captured images, the movement of an object makes the 
derivation of correspondences between images of an image sequence extremely 
difficult or impossible. Though it would be apparent to one of ordinary skill in the art, the 
method of Davidson et al. does not provide a means to remedy this potential problem. 

Le Roux, et al. describe several methods to differentiate and extract regions 
corresponding to moving objects in video images (Le Roux et al. Introduction). 

The teachings of Le Roux et al. and Davidson et al. are combinable because 
they are analogous art. In particular, the teachings of both Le Roux et al. and Davidson 
et al. are directed toward the processing of a sequence of images. Therefore, it would 
have been obvious to one of ordinary skill in the art, at the time of the applicant's 
claimed invention, to use moving object segmentation, according to any of the various 
techniques discussed by Le Roux et al., to extract regions corresponding to moving 
objects from the collection of images used by the method of Davidson et al. to generate 
a 3D model. The motivation for doing so would have been to eliminate regions 
corresponding to moving objects from the collection of images. By eliminating these 
regions, correspondences, and hence a 3D model, can be derived for regions of these 
images that do not move. Combining the teachings of Le Roux et al. and Davidson et 
al., in this manner, yields a 3D modeling method that conforms substantially to that of 
claim 10. 
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Allowable Subject Matter 

1 3. Claims 3 and subsequent dependent claims 6, 7 and 8 are allowed. 
The following is a statement of reasons for the indication of allowable subject 

matter: 

The following is in regard to Claim 3. As mentioned above with respect to claims 
1-2 , the method obtained by combining the teachings of Davidson et aL and 
Azarbayejani et al., involves assigning both a weight and resolution to pixels of the 
current image. Though one could arbitrarily designate these values as collectively 
constituting a relevance value, this does not follow directly or impliedly from the 
teachings of Davidson et al. or Azarbayejani et al. or, for that matter, any encountered 
prior art methods. Despite this, it would be apparent, if one were to devise such a 
reference value, that in order for it to be meaningful - or, more specifically, for it to 
denote the relevance of pixels - within the context of the method obtained by combining 
the teachings of Davidson et al. and Azarbayejani et al. - the relevance value should 
be directly proportional to the resolution. This follows from the teachings of Davidson et 
al. (Davidson et al. Fig. 44a, steps S558-S560). Taking this into account, selected 
pixels, which according to Davidson et al. have highest 3D resolution (pixels with SHIFT 
□ 10% of object size), would, in turn, have the highest relevance value among all image 
pixels. Providing an indication (e.g. a table) indicating which pixels are selected would 
be inherent to the process of selection. Such a collection can be considered a mask. 
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Though these latter aspects of claim 3 follow implicitly from a meaningful notion of 
relevance value within the context of the method above, it cannot be said that these 
aspects would be readily apparent from the teachings of Davidson et al. or 
Azarbayejani et al. or any encountered prior art methods. In this manner, the subject 
matter of claim 3 is unique and non-obvious with respect to the encountered prior art 
methods. 

The following is in regard to Claims 6-8. In and of themselves, the substantive 
limitations set forth in claims 6-8 introduce nothing over similar prior art methods. 
However, by virtue of their dependence on allowable claim 3 (see below), these claims, 
in turn, propose allowable subject matter. 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Wes Tucker whose telephone number is 571-272-7427. 
The examiner can normally be reached on 9AM-5PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on 571-272-7429. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Wes Tucker 
1-8-05 



